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THEORIES OF STUDENT SUCCESS: 



EVALUATING THE EFFECTIVENESS OF AN INTERVENTION STRATEGY 

Many first and second year college students experience disappointing academic 
performance, frequently putting them at risk of academic probation. The institution in which this 
study was implemented places a student on academic probation when the grade point average 
(GPA) drops below a 2.0 or the student experiences two consecutive academic semesters with 
GPAs below 2.0, regardless of the overall GPA. Since these circumstances negatively influence 
student persistence, institutions have commonly developed various interventions to address 
student attrition. In this case, students placed on probation are contacted by the University and 
offered the opportunity to enroll in a three-credit hour course, Theories of Student Success (TSS). 
This study evaluates the effectiveness of the TSS intervention strategy. 

Existing literature pertaining to interventions addressing student attrition and retention in 
higher education is quite extensive. Some of the more common interventions include: peer 
mentoring (Terrion and Leonard, 2007), curriculum development (Taylor, 2005; Sauer and 
O’Donnell, 2006), one-on-one counseling with student development advisors and/or faculty 
(Kadar, 2001; McArthur, 2005; Bishop, 2006), seminars for first year students (Raymondo, 
2003), and a variety of programs intended to integrate students academically and socially within 
the contexts of the institution; most of which are commonly based on Tinto’s model of academic 
and social integration (Tinto, 1975; 1982; 1993). However, the body of literature pertaining to 
these interventions focuses primarily on the need and rationale for such practices much more 



than the evaluation of them. 




There is a growing call to incorporate institutional research (IR) as a means of 
accountability of and support for retention and academic success (St. John, 2006; Hossler, Kuh, 
and Olsen, 2001). Hossler, Kuh, and Olsen (2001) utilized institutional research to supplement 
literature guided efforts in developing strategies to better prepare first year students for academic 
success at Indiana University Bloomington. St. John (2006) argues that institutional research can 
provide much needed accountability, strengthening educational policies and programs. 

Therefore, while this paper adds to existing empirical research pertaining to retention and 
success of first year students, it also uses IR to assess the effectiveness of a program 
implemented as an intervention strategy addressing student attrition. Additionally, the project 
makes a methodological contribution with the use of the Rasch measurement model for analysis, 
as this model provides more reliable analysis than traditional analytical models (Bradley and 
Sampson, 2005; Royal, 2008). 

Intervention Strategy and Purpose 

Theories of Student Success (TSS), a 3 credit-hour graded course designed for 
undergraduate students interested in increasing their knowledge of academic life and improving 
their ability to achieve academic success is offered to first year students on academic probation. 
The objective of the course is to introduce and/or strengthen the understanding of higher 
education to students who are on academic probation. Several essential elements of academia are 
emphasized, including theories of student development, the sociology and history of higher 
education, and the organizational structure of post- secondary institutions. Particularly, the course 
attempts to answer such questions as: What is my learning style and how should I adjust to 
accommodate college work? What does it take to be successful in college? What are some of the 




issues facing college students on today’s campuses? What do professors expect and how do 
successful students meet those expectations? 

The purpose of this study was to evaluate the effectiveness of Theories of Student Success 
as an intervention strategy designed for students on academic probation. A quasi-experimental 
design was generated to identify probationary students’ sentiments about: 

• their first-year experience; 

• attitudes and perceptions about faculty, advisors and the institution; 

• knowledge of institutional services afforded to them; 

• sense of self (including personal growth and academic/social priorities); and, 

• various personal and professional support mechanisms. 

Treatment and control group samples were compared to determine how students in the treatment 
group fared in comparison to the control group with regard to specific items within the 
aforementioned constructs. 

Methods 

Population and Sampling 

A list of email addresses for freshmen on academic probation as of the Fall 2006 
semester was provided by the University’s Office of Assessment. Upon receipt of the data the 
primary investigator cleaned the dataset and separated students according to two criteria: those 
on academic probation who were currently enrolled in regular courses, and those who were 
taking regular courses but also enrolled in TSS. These two groups served as a control group and 
a treatment group for this study. 

Because a comprehensive list of students was available a census sample was employed. 
All students enrolled in the three sections of TSS were given a paper-pencil questionnaire at the 
end of class and asked to voluntarily complete it. A total of 76 questionnaires were administered 
and 69 students completed the survey, thus resulting in a 91% response rate. A census sample 




was also employed for the general probation group. A web-based survey was utilized because the 
researchers believed most college students regularly access their email accounts. In total, 581 
students were sent an email (with two reminders) asking for their participation in the study. Of 
those 581 students, 198 returned completed questionnaires. This resulted in a 34% response rate, 
which is substantially higher than the typical 15-20% response rate of undergraduate students at 
this institution. 

Instrumentation 

The questionnaire was adopted from a 1998 study by Fernandez, Whitlock, Martin and 
VanEarden which evaluated students’ first year experience. The original questionnaire was 
modified to fit the needs of this study, which included several additional items and demographic 
questions. 

The instrument used for the control group consisted of 27 primary items, 18 demographic 
and four open-response items. The instrument given to the treatment group consisted of 32 
primary items, 19 demographic and five additional open-response items. Students were asked to 
provide their level of agreement with these 28 or 30 items utilizing a four point Likert-type scale 
as follows: Strongly Agree = 4, Agree = 3, Disagree = 2 and Strongly Disagree = 1. Additional 
demographic questions were asked to gain a better understanding of the composition of the 
students in both groups. Open-ended response items were included, allowing students to expand 
on previous answers and to offer information that was otherwise unsolicited. 

Because of the comparative nature of this evaluation study, some questions were 
carefully worded differently on the two questionnaires to distinguish the TSS class experience 
and its instructors from other University class experiences and instructors. An example would 
include: “Generally speaking, I believe University faculty are supportive of me” and “I believe 




my TSS instructor is supportive of me”. Although it is true the same construct is not being 
measured in comparative questions like these, these items do allow for direct comparisons 
between the two constructs. Overall, 26 items were produced that could be compared to the other 
group. An additional 6 items pertained to specific items relating to one particular group. 

Data Analysis 

Data were collected using Zoomerang survey software. Descriptive statistics were 

generated using Minitab. Rasch measurement, a form of item response theory, was utilized as the 

primary method of data analysis because the technique provides a number of advantages over 

traditional statistical approaches. For example, Bradley and Sampson (2005) state: 

Whereas the classical model [traditional statistics] produces a descriptive summary based 
on statistical analysis, it is limited, if not absent, in the measurement capacity. In contrast, 
Rasch measurement tackles many of the deficiencies of the classical test model in that it 
has the capacity to incorporate missing data, produces validity and reliability measures 
for person measures and item calibrations, measures persons and items on the same 
metric, and is not dependent on the particulars of the sample. Applications of the Rasch 
model allow the researcher to identify where possible misinterpretation occurs and which 
items do not appear to measure the construct of interest, while producing information 
about the structure of the rating scale and the degree to which each item contributes to the 
construct. Thus, it provides a mathematically sound alternative to traditional approaches 
to survey data analysis (p. 13). 

Winsteps measurement software was used to carry out the Rasch analysis. 

Survey Validation 

Often referred to as “validating a survey”, Rasch measurement determines the fit of the 
data to the Rasch model, as well as determines how well the rating scale is functioning. Survey 
validation asks, “How well are the items measuring what they are intending to measure?”, “Are 
all respondents interpreting the questions similarly?”, and “Are the response options sufficient?”, 
meaning should response options be collapsed and limited to fewer response categories, or 



should respondents be given additional categories to more accurately capture their sentiments? 




Having strong theoretical underpinnings and carefully avoiding methodological assumptions is 



essential for quality research; therefore, this study ensures many of the methodological 
assumptions of traditional survey research (i.e., treating ordinal scales as interval, treating raw 
scores as measures, applying linear statistics to nonlinear data, etc.) were overcome via the use of 
the one -parameter Rasch measurement model. 

Results and Discussion 
Demographic Information for Respondents 

The sample for this study consisted of 198 General Probation students and 69 TSS 
Probation students. Demographic information is presented below for both the General Probation 
group (see Table 1) and the TSS group (see Table 2). 

With regard to the collective sample, both groups contain 80% White students, a statistic 
that is reflective of the University’s campus. There is some variation amongst the sex variable, as 
the general probation sample is nearly even and the TSS sample is approximately two-thirds 
male. 



Table 1 

Demographic Characteristics of General Probation Group (n = 198) 



Characteristic 


n 


% 


Sex 


Male 


103 


52 


Female 


95 


48 


Race 


White 


159 


80 


Black 


21 


11 


Hispanic 


5 


3 


Asian or Pacific Islander 


4 


2 


Other 


9 


5 






Table 2 

Demographic Characteristics ofTSS Probation Group (n = 69) 



Characteristic 


n 


% 


Sex 


Male 


44 


64 


Female 


25 


36 


Race 


White 


54 


80 


Black 


11 


16 


Hispanic 


1 


1 


Asian or Pacific Islander 


1 


1 


Other 


2 


2 



Reliability, Model Fit and Rating Scale Functioning 

Winsteps measurement software reports reliability estimates for both persons and items. 
For the General Probation group, person reliability was .97 and item reliability was .91. For the 
TSS group, person reliability was .89 and item reliability was .85. Both samples indicate 
moderate -high reliability measures for both persons and items. Tests for data- model fit 
confirmed data fit the Rasch model, thus confirming Rasch measurement an appropriate 
technique for data analysis. Tests for rating scale functioning verifying appropriate response 
options were provided by way of probability curve analysis and rating scale diagnostics. 
Summary of Rating Scale Diagnostics 

Responses to survey items correspond to the aforementioned four-point Likert-type rating 
scale, where: Strongly Agree = 4, Agree = 3, Disagree = 2 and Strongly Disagree = 1. Advancing 
average measures and step calibrations ensure the rating scale measures are stable and accurate. 
The tables below indicate respondents are utilizing the full range of the scale; however, the 







majority of responses are Agree/Strongly Agree statements. This indicates most survey items 



were likely to be endorsed or agreed upon by the average respondent. 



Table 3 

Summary of Rating Scale Diagnostics for General Probation Sample 



Category 


Observed 
Count (%) 


INFIT 
Mean Sq. 


OUTFIT 
Mean Sq. 


Strong Disagree 


190 (4%) 


1.11 


1.16 


Disagree 


1244 (23%) 


1.00 


1.06 


Agree 


2963 (55%) 


.92 


.91 


Strongly Agree 


949 (18%) 


.98 


.98 



Note : Category, observed count, and percentage indicate the numbers of respondents who chose 
a particular response category, summed for each category across all items. 

Table 4 

Summary of Rating Scale Diagnostics for TSS Sample 



Category Observed INFIT OUTFIT 

Count (%) Mean Sq. Mean Sq. 



Strong Disagree 


44 (2%) 


1.01 


1.06 


Disagree 


312(14%) 


1.05 


1.07 


Agree 


1297 (60%) 


.95 


.90 


Strongly Agree 


518 (24%) 


.98 


.97 



Note : Category, observed count, and percentage indicate the numbers of respondents who chose 
a particular response category, summed for each category across all items. 

Empirical Characteristics of Responses 

The control and treatment group responses were compared via an item map (see Figure 1) 
to detect meaningful differences between the two groups. Results indicated students in TSS 
reported higher ratings on the vast majority of primary items. Specific to treatment versus control 
group comparisons, results revealed TSS students indicated the course helped... 

• develop a commitment to personal achievement 

• cultivate a sense of responsibility for their own behavior 

• develop the capacity to make wise decisions 








• improve their self-esteem/self-confidence 

• instill a sense of discipline within them 

• motivate them for academic success 

• provide insights about college they probably would not receive elsewhere 

• play a significant role in helping them discover their personal strengths and weaknesses 

• play a significant role in helping them overcome their academic difficulty 

...more so than students from other courses without a treatment. 



Figure 1 

Response Hap Comparison. 
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Table 5 displays a list of corresponding survey items to the abbreviations in the table. 



Table 5 
Survey Items 



Item ID 
1 
2 

3 

4 

5 

6 

7 

8 
9 

10 

11 

12 

13 

14 

15 

16 

17 

18 

19 

20 
21 
22 

23 

24 

25 

26 



Item 

Overall, my first-year experience at [this university] was positive 
My courses at [this university] [TSS] helped me develop a commitment to 
personal achievement* 

My courses at [this university] [TSS] helped cultivate a sense of responsibility for 
my own behavior* 

My courses at [this university] [TSS] helped me develop the capacity to make 
wise decisions* 

My courses at [this university] [TSS] helped me to mature as an individual* 

My courses at [this university] [TSS] helped improve my self-esteem/self- 
confidence* 

I became friends with other students in my classes 
I became friends with students who were not in my classes 
I enjoyed the social aspects of my first year 
I know where to find support on campus for academic problems 
I know where to find support on campus for personal problems 
I am confident about my abilities to manage my time 
There is a sense of community here at [this university] 

I feel part of a community here at [this university] 

Prior to entering [this university], I felt this was the right school for me 
Having spent a previous semester (or more) here, I believe [this university] is the 
right school for me 

Generally speaking, I believe [this university’s] faculty [TSS] are supportive of 
me* 

I believe my advisor is supportive of me 
My academic life is a high priority 
My social life is a high priority 

My courses [TSS] have helped instill a sense of discipline within me* 

My courses [TSS] have helped motivate me for academic success* 

My courses [TSS] have taught me insights about college I probably would not 
have learned elsewhere* 

My courses [TSS] have played a significant role in helping me discover my 
academic strengths and weaknesses* 

My courses [TSS] have played a significant role in helping me discover my 
personal strengths and weaknesses* 

My courses [TSS] have played a significant role in helping me overcome my 
academic difficulty* 



*Items denoted with an (*) indicate items that differed per survey (General v. TSS) 





The most noticeable difference between the two item maps pertains to item Q17. The 
item reads “Generally speaking, I believe [the university’s] faculty [TSS instructor] are 
supportive of me”. The map of General Probation student lists this item nearly one standard 
deviation below the mean. The TSS map lists the item about 1.5 standard deviations above the 
mean. It is clear that students enrolled in TSS recognized the supportive efforts of those enrolled 
in the course. Research suggests students who view faculty more favorably are more likely to 
persist in courses and perceive the institution as more welcoming and caring (Miller, 2007). 

Item Q18 is also highly endorsed amongst students enrolled in TSS and moderately 
endorsed in the General Probation group. This item reads “I believe my adviser is supportive of 
me”. It is theorized the reason this item is so highly endorsed by the TSS sample is because the 
respective academic advisors referred the students to the TSS class and closely monitored their 
progress. The additional conversations and planning sessions with advisors likely gave these 
students the impression their advisors were more supportive. One might speculate additional 
meetings with advisors would increase this rating for the General Probation sample as well. 

The item with the lowest mark for the General Probation group was item Q6, “My 
courses at [this university] helped improve my self-esteem/self-confidence”. This item fell over 
two standard deviations below the mean on this map. The converse, “My TSS course helped 
improve my self-esteem/self-confidence”, was notably higher on the TSS map as it was within 
one standard below the mean. These differences suggest TSS students experienced increased 



confidence in their academic abilities. 




There are two items in particular that score higher on the General Probation student map 
than the TSS map. They are items Q16 and Ql. Item Q16 reads “Having spent a previous 
semester (or more) here, I believe [this university] is the right school for me”. On the General 
Probation map, this item falls within one standard deviation above the mean. On the TSS map, 
this item falls a full standard deviation below the mean. The authors speculate this difference is 
due to TSS students informed awareness of university life and expectations. This heightened 
understanding of the university atmosphere and culture likely forces students to evaluate their 
role within the institution. Having given it more thought than their General Probation group 
counterparts, TSS are more likely to identify their “fit” both with and within the university. 

The other previously mentioned item, Ql, reads “Overall, my first-year experience at 
[this university] was positive.” General Probation students perceived this item near the mean, 
well within one standard deviation of the mean. TSS students rated this item a tie for the lowest 
overall item with a score that fell over one standard deviation below the mean. Similar to Q16, 
the authors theorize an informed perspective of university life and expectations helped TSS 
students realize the true measuring stick for a positive first-year. The informed understanding of 
what constitutes a positive first year made TSS students more critical of themselves, thus leading 
to a lower score for this item. 

Interestingly, one item that fell below one standard deviation on both maps was item Q12 
which read “I am confident about my abilities to manage my time”. Although significant time 
was devoted to this issue in the TSS class, it appears instructional attempts were insufficient. 
There were no survey items designed to identify how students’ time is used, or how students 
define effective time management. Therefore, the items employed in this survey do not provide 
sufficient data to challenge or confirm responses for Q 12. 




For items appearing exactly the same on both surveys, TSS students reported 
significantly higher scores with regard to knowing where to find support on campus for both 
personal (Q 11) and academic (Q10) problems. This, too, is affirming because identification of 
campus resources and their uses was a notion emphasized in the TSS course. 

One additional element of the TSS course involved mandatory, one-on-one consultations 
with each student every other week throughout the semester. Of all the students surveyed, 93% 
of students found the consultations helpful. Students noted that developing a working academic 
relationship with the instructor positively impacted overall feelings about course work, increased 
efficacy for completing assignments, and provided a sense that faculty members (represented by 
the TSS course instructor) cared about student success. 

Significance of Study, Implications and Future Research 
This study contributes to the research literature in four key ways. First, the study not only 
identifies several important and potentially problematic issues for many first-year students, but 
also provides an empirically-supported instructional model that effectively addresses many of 
these issues and has the potential to significantly improve students’ intellectual, social, and 
emotional experiences at college. Second, the modified instrument and study design serves as a 
useful evaluation model for other institutions/programs who are concerned with evaluating the 
effectiveness of academic probation programs. Third, the study serves as a methodological 
framework for related studies, as the Rasch model minimizes common erroneous assumptions of 
survey research and provides a more valid and reliable approach to analyzing survey data. 
Further, the use of items maps can prove to be an invaluable asset to researchers looking for both 
“big picture” and comparative results. Finally, the findings of this study warrant further 




investigation of issues relating to students on academic probation and additional applications of 
institutional research. 

While the findings from this study support the TSS course as an effective intervention for 
first year students on academic probation, replication of the project is advisable to more fully 
identify what aspects of the course are most beneficial and which elements are in need of 
adjustment. Additionally, each generation of students presents new needs and issues, challenging 
the findings of previous groups. Therefore, the instrument will need to be re-assessed for future 
applications. 
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